Why is a linear model less than ideal for this data?
Modelling binary responses
Orange line (logistic model fit) is similar to computing a running average of the 0s/1s. It’s much better than the linear fit, because it remains between 0 and 1, and can be interpreted as proportion of 1s.
What is a logistic function?
The logistic function
Instead of predicting the outcome directly, we instead predict the probability of being class 1, given the (linear combination of) predictors, using the logistic function.
the formula can be extended, using dummy variables.
follows from the above, extended to provide probabilities for each level/category, and the last category is 1-sum of the probabilities of other categories.
the sum of all probabilities has to be 1.
Connection to generalised linear models
To model binary data, we need to link our predictors to our response using a link function. Another way to think about it is that we will transform \(Y\), to convert it to a proportion, and then build the linear model on the transformed response.
There are many different types of link functions we could use, but for a binary response we typically use the logistic link function.
Interpretation
Linear regression
\(\beta_1\) gives the average change in \(Y\) associated with a one-unit increase in \(X\)
Logistic regression
Because the model is not linear in \(X\), \(\beta_1\) does not correspond to the change in response associated with a one-unit increase in \(X\).
However, increasing \(X\) by one unit changes the log odds by \(\beta_1\), or equivalently it multiplies the odds by \(e^{\beta_1}\)
Maximum Likelihood Estimation
Given the logistic \(p(x_i) = \frac{1}{e^{-(\beta_0+\beta_1x_i)}+1}\) choose parameters \(\beta_0, \beta_1\) to maximize the likelihood:
fit <-glm(default~balance, data=simcredit, family="binomial") simcredit_fit <-augment(fit, simcredit,type.predict="response")
Orange points are fitted values, \(\hat{y}_i\). Black points are observed response, \(y_i\) (either 0 or 1).
Fitting credit data in R
We can use the glm function in R to fit a logistic regression model. The glm function can support many response types, so we specify family="binomial" to let R know that our response is binary.
fit <-glm(default~balance, data=simcredit, family="binomial") simcredit_fit <-augment(fit, simcredit,type.predict="response")
# A tibble: 2 × 4
# Groups: default [2]
default No Yes Accuracy
<fct> <int> <int> <dbl>
1 No 9625 42 0.996
2 Yes 233 100 0.300
Compute the balanced accuracy.
Unbalanced data set, with very different performance on each class.
How good is this model?
Explains about half of the variation in the response, which would normally be reasonable.
Gets most of the smaller but important class wrong.
Not a very useful model.
A warning for using GLMs!
Logistic regression model fitting fails when the data is perfectly separated.
MLE fit will try and fit a step-wise function to this graph, pushing coefficients sizes towards infinity and produce large standard errors.
Pay attention to warnings!
logistic_fit <- logistic_mod |>fit(default_new ~ balance, data = simcredit)
Warning: glm.fit: algorithm did not converge
Warning: glm.fit: fitted probabilities numerically 0 or 1
occurred
Discriminant Analysis
Linear Discriminant Analysis
Where would you draw a line to create a boundary separating Adelie and Gentoo penguins?
Where are the sample means?
What is the shape of the sample variance-covariance?
Linear discriminant analysis assumes the distribution of the predictors is a multivariate normal, with the same variance-covariance matrix, separately for each class.
Assumptions underlie LDA
Source: https://xkcd.com
All samples come from normal populations
with the same population variance-covariance matrix
LDA with \(p=1\) predictors 1/4
If \(K = 2\) (two classes labelled A and B) and each group has the same prior probability, the LDA rule is to assign the new observation \(x_0\) to class A if
\[
x_0 > \frac{\bar{x}_A + \bar{x}_B}{2}
\]
It’s a really intuitive rule, eh?
It also matters which of the two classes is considered to be A!!!
So maybe easier to think about as “assign the new observation to the group with the closest mean”.
How does this rule arise from the assumptions?
Bayes Theorem 2/4
Let \(f_k(x)\) be the density function for predictor \(x\) for class \(k\). If \(f\) is large, the probability that \(x\) belongs to class \(k\) is large, or if \(f\) is small it is unlikely that \(x\) belongs to class \(k\).
According to Bayes theorem (for \(K\) classes) the probability that \(x\) belongs to class \(k\) is:
where \(\mu_k\) and \(\sigma^2_k\) are the mean and variance parameters for the \(k\)th class. We also assume that \(\sigma_1^2 = \sigma_2^2 = \dots = \sigma_K^2\); then the conditional probabilities are
A simplification of \(p_k(x_0)\) yields the discriminant functions, \(\delta_k(x_0)\):
\[\delta_k(x_0) = x_0 \frac{\mu_k}{\sigma^2} - \frac{\mu_k^2}{2 \sigma^2} + log(\pi_k)\] from which the LDA rule will assign \(x_0\) to the class \(k\) with the largest value.
Let \(K=2\), then the rule reduces to assign \(x_0\) to class A if
A \(p\)-dimensional random variable \(X\) has a multivariate Gaussian distribution with mean \(\mu\) and variance-covariance \(\Sigma\), we write \(X \sim N(\mu, \Sigma)\).
NOTE: Class A and B need to be mapped to the classes in the your data. The class “to the right” on the reduced dimension will correspond to class A in this equation.
Computation
Use sample mean \(\bar{x}_k\) to estimate \(\mu_k\), and
to estimate \(\Sigma\) use the pooled variance-covariance:
\[
S = \frac{n_1S_1 + n_2S_2+ \dots +n_kS_k}{n_1+n_2+\dots +n_k}
\]
Example: penguins 1/3
Summary statistics
# A tibble: 2 × 3
species bm bd
<fct> <dbl> <dbl>
1 Adelie 3701. 18.3
2 Gentoo 5076. 15.0
Discriminant space: LDA also provides a low-dimensional projection of the \(p\)-dimensional space, where the groups are the most separated. For \(K=2\), this is
\[
\Sigma^{-1}(\mu_A-\mu_B)
\]
The distance between means relative to the variance-covariance, ie Mahalanobis distance.
Discriminant space
The dashed lines are the Bayes decision boundaries. Ellipses that contain 95% of the probability for each of the three classes are shown. Solid line corresponds to the class boundaries from the LDA model fit to the sample.
(Chapter4/4.6.pdf)
Discriminant space: using sample statistics
Discriminant space: is the low-dimensional space (\((K-1)\)-dimensional) where the class means are the furthest apart relative to the common variance-covariance.
The discriminant space is provided by the eigenvectors after making an eigen-decomposition of \(W^{-1}B\), where
\[
B = \frac{1}{K}\sum_{i=1}^{K} (\bar{x}_i-\bar{x})(\bar{x}_i-\bar{x})^\top
~~~\text{and}~~~
W = \frac{1}{K}\sum_{k=1}^K\frac{1}{n_k}\sum_{i=1}^{n_k} (x_i-\bar{x}_k)(x_i-\bar{x}_k)^\top
\]
Note \(W\) is the (unweighted) pooled variance-covariance matrix.
Mahalanobis distance
For two \(p\)-dimensional vectors, Euclidean distance is
\[d(x,y) = \sqrt{(x-y)^\top(x-y)}\]
and Mahalanobs distance is
\[d(x,y) = \sqrt{(x-y)^\top\Sigma^{-1}(x-y)}\]
Which points are closest according to Euclidean distance? Which points are closest relative to the variance-covariance?
Discriminant space
In the means of scenarios 1 and 2 are the same, but the variance-covariances are different. The calculated discriminant space is different for different variance-covariances.
Notice: Means for groups are different, and variance-covariance for each group are the same.
Quadratic Discriminant Analysis
If the groups have different variance-covariance matrices, but still come from a normal distribution
Quadratic DA (QDA)
If the variance-covariance matrices for the groups are NOT EQUAL, then the discriminant functions are:
where \(\Sigma_k\) is the population variance-covariance for class \(k\), estimated by the sample variance-covariance \(S_k\), and \(\mu_k\) is the population mean vector for class \(k\), estimated by the sample mean \(\bar{x}_k\).
Quadratic DA (QDA)
A quadratic boundary is obtained by relaxing the assumption of equal variance-covariance, and assume that \(\Sigma_k \neq \Sigma_l, ~~k\neq l, k,l=1,...,K\)
true, LDA, QDA.
(Chapter4/4.9.pdf)
QDA: Olive oils example
Even if the population is NOT normally distributed, QDA might do reasonably. On this data, region 3 has a “banana-shaped” variance-covariance, and region 2 has two separate clusters. The quadratic boundary though does well to carve the space into neat sections dividing the two regions.
Checking the assumptions for LDA and QDA 1/2
Check the shape of the variability of each group could be considered to be elliptical, and the size is same for LDA but different to use QDA.
